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Section I 

t 

Introduction 



f > 



. The Diagnostic Testing System' (DX) is an integral system for 
developing and administering tests. The • system can be utilized for testing 
in any subject ngiter, or any number of subject matters, at any level ^ri 
scholastic or cognitive continuums. The ma jar, purpose of DX is to provide 
diagnostic about the level at^iich a given student (or group of students) 
comprehend the material or concept being tested. , 

Based on a one-disk or two-disk configuration of the Apple II (or 
^ Apple II+), with 48k of memory, the DX package is^an interactive, diagnostic 
o system for multiple -choice testing. It will provide ypu with a way- to 
enter files of multiple -choice questions, to make changes in the files, to 

examine existing records of performance, and to administer multiple -choice ' 

, • . - • ' ' f r • 

diagnostic tests by computer to individual students. Basically, as the 

■ respondent or student responds to a set of tesfitems the °DX system is 

recording the performance. Simultaneously, DX is preparing an , t \ 
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interpretive, diagnostic readout for the instructor which includes a record 
of the performance' as well as a shor^Pftarrative explaining the level at 
which the respondent comprehends.^he material tested. - a 

*The DX package includes eight subprograms. These component programs 

include test item writing and editing systems, a test outcome, feedback - 

^ ' —■ 

■ system, and a system for determining how many and which tests will be >t 
administered at a given time. DX, in its entirety, is-designed so as to ^ 

present a technically simple paickage which can be easily learned and put 

' '' -r I ' * 

into operation by (/lassroom tekhers, ( including those with no^pomputer 

know-how whatsoever, if /ou c|n type, you can use the DX package. 

■ • m •■" • . . \ • • • ; 



# Thi* nanual is organized into eight sections, including this . 
Introduction. The theoretical background which underlies the DX method is 
(fescMberf"TFK Section II. The manual covers various strategies for def ining - 
and putftin^g into operation the concept of ".levels of comprehensioniMrfhich DX 

isigned to assess. Section jIII delineates the lattice whicfc is the 
basis] for item sequencing in the DX test. It is on the basi^ of this item 
order that the program is eventually able to deliver a diagnostic 
assessment of pupil comprehension. 

The fourth, fifth* and sixth sections acquaint the user with the 
^software included in the DX package. In Section IV, each of the 

subprograms is described in detail. In Section V*, entire DX test 

'- * . ■ • . '. • * 

development process is illustrated by reviewing an example, test file (which 

is included in your program disk). Finally, Section VI provides hands-on, 

guided demonstration by which the user creates an original item set and 

test program based on -^he DX lattice foundation. ' » 

Section VII illustrates the nature of the feedback whith the DX 

package produces, as well as procedures for interpreting and scoring DX 

outcomes. Test results are reported in three coordinated, simultaneous 

statements which provide the instruc^or^with information concerning each 

respondent's performance on the DX test. Thesedata can be analyzed either 

by traditional metrtods.of number of correct responses, or by level- of 



trtods. of nu 
iescribed i 



comprehension as described in Sections II and III. The final section 
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(VIII) provides concluding remarks for the manual. 



Section II 
Theoretical Background 



* Recent research by Vtefsrs and others has shown that the use of computer 

managed adaptive testing pan be linked effectively and efficiently with 

item response theory. The increasing availability of s^tll computers. makes 

the use of sophisticated models of objective measurement based^ on the 

interaction of individlia/ examinees with single items feasible in a wide 

„. ' . 

variety of training and testing applications. At the Center for the Study 
/ 

Of Evaluation, University of California, Los Angeles, we have been 

• * * 

investigating the advantages of computer prompted interactive testing with 

Vs * • • • 

nteht of 'analyzing and diagnosing patterns of errors. 

The first volume of Journal, of Educational Research contained a study 
of diagnosis of error types (Willing, 1920); the' first volume of Journal of 
General Psychology contained a major article by Spearman (1928) on the ; - 
"Origin of error";, the secor\d volume of the British Journal of Educational 
Psychology presented a lengthy analysis of theories of cognitive error 
'(Fortes, 1932). Despite the intervening years, the concepts about error 
initiated in those articles hold up surprisingly welW-Em>rs tend to show 
themselves as matters of either principle (such as faulty reasoning, v 
misunderstanding, or inability to apply a correct method or 'strategy ) or^ 
accuracy (such as .errors in copying, manipulating numbers, or misplacing 
parts^ of the problem). 

' The Tatsuokas have been studying a complex series of erroneous rules 
of operation spe^tfic to the task of subs traction. Other researchers, 
using latent trai^ models (Same jima, Wright, and lord, for example)), have 
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treated incorrect responses on a test aSstemming in part from chance. We 

believe that there are a number of important diagnostic hypotheses about 

patterns of error relating to the specific individual -feeing tested. These 

.have to do "with the student's Ability to determine the correct answer 

■bef^re^prompting or after either direct or indirect prompting 4 , learning of 

the initial concept might have failed for the student; the underlying 

conceptual grasp may be -faulty'; or, on the other hand\, a short detour 

through less difficult material may allow the' student \o succeed on. retry 
*•* " * 

with a more difficult higher order question. For these hypotheses itTs 

appropriate to collect detailed evidence'by further prompts from the same 
item, by "decompressing" or "dismantling" the, item in obtaining parallel 
responses to its component parts, or by* administering one or more^parallel 
x items. Evidence from. a fin'ite number of different items or of components 
of decompressed items, gathered on the same individual,.!'? needed to 
establish £he credibility of each hypothesis. Such, credibility is 
necessary in either of two circisiistances^-developnent of detailed . 
^diagno|,tic reports concerning stents and in automatic modification of the 
testing and instructional prograr 



For atestV reasonable lenrrgth, the number of possible sets of 
hypotheses that coul cDexplain a particular response pattern would be 
\ astronomical. However, by planning that the testing procedure will remain 
within a^pecific pool of items, we can limit the number of sets to be 
processing capacity of the compute r,> allows the, scanning of response 
patterns in order to determine "likely" hypotheses during the adaptive 
testing session itself. / 

There are- several^views about v the issue of levels within 
multiple-jchpice testing. One widely regarded approach stems from work 



conducted by Benjamin Bloom and colleagues during the 1950' s (Bloom, 
Englehart, Furst, Hill, S krathwohl , 1956). Recent work by a number of 
researchers ha^ reexamined the^xoncepts of "taxonomy" of skills of 
thinking. (The interested reader is referred to Carroll ,Cl978, for a; • 
review of seven separate approaches to this problem.) Two books have been 
recently written on the tasks of writing good items for multiple -choice 
testing; one is the easily readable book by Miller, Williams, A Haladyna, 

1978: Beyond Facts: Objective Ways *to Measure Thinking. The more 

\ . ' ^ ^ ^ 

thorough of the two is the most recent contribution to this area, A 

Technology for -Test-Item Writing by Roid & Haladyna (1982). 

Williams and Haladyna have developed a three-dimensional u^del for 

classifying objectives. The first two dimensions in their scheme' are 

alreacfy accounted for by the very nature of the DX testing system: 

Response Mocfe (selected vs. constructed) and Content (facts vs. concepts 

vs. principles). The third dimension of their model provides an extremely 

useful and operational ized taxonony* of levels for cogiitive -tasks. Those 

levels are as follows (Roid * Haladyna, 1982, pg. 1 165-166): 



1. Reiteration is 



a task which requires the ^student to 



Apt / i 

recognize or produce information in essentially the 

same form in which it was presented. 
2. . Summarization requires the student to recall the 
substance of the information without a verbatim 
requirement. 

Illustration requires the student to show comarehwv^ion 
by recognizing or constructing illustrative pxamples 
based on material not contained Ms the source message. 



\ 

\ v 
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4. Prediction . Using principles 6r rVles derived br 
inferred from the message^jeaterial, the student 
predicts outcomes given previously unmeirtioned or 
unencountered situations*. .. 

fc » * ' . 

5. Evaluation involves (feci son making based on the use of 

, * . 

some criteria. r > v 
- 6. Application. Using all previous levels of the 

taxonomy, the studen£ is able ^derive a system for if 

prescribingv the necessary precedents and prerequisites 

for achieving a desired outcome. 
The Williams and Haladyna system offers the teacher or test builder 
several options for test item construction. Forvefcpple, one could choose 
the first three leve/ls ornhe taxonomy. The high level, most dffficult 
test items might require the students to explain *why a given fact or eveht 
occurs, or identify and classify stimuli according to some underlying 
criteria (Illustration). The moderately difficult items could requirje the 

' • * 

recognition of non-verbatim restatements of the information message or 
underlying tqpjc (Summarization).. The least difficult, lowest level test 
ttems' might simply require verbatim recognition (Reiteration). 

eiearlir, there are numerous other ways one might utilize the Roid and 

» ** 

Haladyna taxonomy in the DX system. These should be explored .further by 
each interested test developer and made relevant to the specific tojfic and 
audience of the test. * * 

Another extremely promising classification system for guiding item 
development is the Solo taxonomy (Collis & Briggs, 1979). This system was 
based specifically upon the' logic-of the successive- levels* in the 
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attaiahent of cognitive reasoning abilities described by developmental. 




researchers (cf. Piaget 4 Cqllis, 1975). At each developmental level there 
are three possible response modes provided, which are easily translated 
into item types or Miff iculty levels (pg. 15-20): 

' IT » 

1. Capacity involves the simple reflection of ^information, 
although not necessarily in perfbct verbatim fashion. 
This involves, the employment of the minimum number of % 
pieces of information in order to respond. 

2. Relating bperatibn allows the student to identify an 
overall concept or principle which accounts for the 

i information provided. This goes beyond the restatement 
or verbatim recall of ttte materia^. 'More sophisticated 
Reasoning is required, and the student must be able to . * xr- 
Teco^viK or (derive by induction some general ■- , 
conclusion (Concerning the information. , 

> 0 . He-re the respondent must 
^n light of other facts known 
of th& message. . This involves 
skills such aV abstraction, juxtaposition, and other ^ 
higher level, inferential cognitive capabilities. The 
student must be able to generalize the informatio|| i 
beyond the setting giA<en, yet not contradict either 
that specific information or other known phenomena, 
The sample item set 'found in this manual uses, a logic similar to the 
Solo taxonomy. Essentially, the lowest jlevel items are restatements of the - 
topical definition! Middle-level items could also be answered by a student # 
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4 ' ' * 

•if the definition ware known, 'but only if fhe student could "read between 

the lines" and recognize the underlying principles anji 



elements operating 



in the definitional message. At the highest level there' was sijpply nofc 
enough information in the' definition alone to allow for correct 
responding." At this most difficult leve'l the student had to go beyond and 
apply 'the; information provided in the definition. 

^^Orfce one becomes acquainted with .the DX system, and after generating 
item sets, one will find no difficulty in becoming quite expert at 
following a three-level taxonomy philosophy. Soon you will be generating 
four 'cfwn levels wjth success and encouraging results. 





ERIC 




Section III 

The Lattice 


* 

The DX interactive diagnostic testing package is based on a 
hierarchical iattijce of eight multiple -choice items. * Each topic to be 
tested is composed of a separate lattice of eight unique items. The 
lattice itself operates on a simple idea of three levels ^of difficulty 
based pn the ta^onomical arguments presented in the previous section. 
There are three response possibilities at each difficulty level. Figure 1 
illustrates the complete lattice. • Note that the informative definition is 
provided as a ninth lattice member. 

' This is how the DX system operates based on the lattice. (Follow 
Figure 1 while you read this. The consequences of correct answers are . 
shown .as solid lines; of incorrect answers, dashed lines.) 

The student is introduced to the first multiple -choice 
\ item in the topic beting tested. The first item is at the 4 
highest, most difficult lattice level: level 3. If the 
student answers the initial question correctly, the computer 
interprets the performance to mean that the student knows 
the topic area and could answer all of the remaining items 
correctly if attempted. The computer then automatically 
introduces the student to the first item in the next topic 
lattice. 

If a student should fail the initial item, the computer 
informs the student of the correct response and then 
proceeds to the next level of the matrix. The -second 



response opportunity is the first item at leve.l 2, the 
moderate -difficulty level. A correct response to £his item 
offers the student a second attempt at the highest level. A 
failure at the midjdle le>/el item sends the student to the | 
lowest level of difficulty.. . 

In the three -level lattice there are 18 discrete paths 
possible. Eleven* response paths lead to failure at onej 
^difficulty level or another. Seven lead to a successful 
completion at the highest level, interpreted as success in 
the topic being tested. Each of the 18 paths clearly \- 
represents a different response pattern." Therefore, the 
computer is programmed to interpret each pattern uniquely 
and deliver a separate diagnostic interpretation. 
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Figure X 



ENTRY 



A diagnostic item lattice for a question set comprised 
of three.levels of difficulty. . Solid lines show the paths 
taken after a correct response; dashed lines show the paths' 
following an incorrect response. 



next question set 



Level 2 




Level 1 



Q 3c failure at level 3 



Q 2c failure at level 2 

I 

! 

Q ^ •! failure at level 1 

t\ 



Although any one student will only take one\pa.th, 
eighteen paths are possible:* 



— next question set 



P 
F 
F 
F 
F 
F 
F 
F 
F 
F 
P 
P 
P 
F 
F 
F 
F 



F 
P 
P 
P 



P 
P 
F 
F 
F 
F 
F 
F 
F 
P 
P 
F 

Def 
Def 
Def 
Def 



next 

P — next 



\ 



P P 
P F 
F 



next 



F Def 
Def 
Def 
Def 



F 
F 
F 
P 
F 



next 



— next 



P 
P 
F 



— next 



level 3 correct on 1st try 
level 3 correct on 2nd try 
level 3 correct on 3rd try 
dx: can't pass level 3 
level 3 correct despite trouble 
dx: erratic; can't pass level 3 
dx: erratic; can't pass level 2 
Level 3 correct w/ definition 
dx: can't pass level 3 w/ defntn 
dx: can't pass level 2 w/ defntn 
dx: can't pass level 1 w/ defntn 
level 3 correct with prompt 
dx: can't pass level 3 w/ prompt 
dx: can't pass level 2 w/ prompt 
level 3 correct with definition 
dx: can't pass level 3 w/ defntn 
dx: can't pass level 2 w/ defntn 
dx: can't pass level 1 w/ defntn 
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*P = pass; F,= failure 
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Section IV 
DX Subprograms 



Within the DX package there are eight subprograms: Menu, I^em File 
Writer, File Modifier, Hard copy, DXOK, Question Index* Copy A, and 
Catalog. All are designed to preclude the need for any previous computer 
training or experience. 'They are self -prompting and generally operate by 
presenting the user with a series of questions and instructions which make^ 

1 

their use simple. ^ 

The first subprogram is MENU. WJien one inserts the disk and turns* on 

the computer, the screen will present, a t-fenu of possible options for your 

» 

use. Each of these options involves one or more separate subprograms. By 
selecting and entering the number associated with a desired Menu item, the 
computer will automatically oblige by presenting the requested subprogram. 

The first Menu choice is for creating a question set. (This operation 
is accomplished by use of the Item File Writer program.) Briefly, this 
program allows the^teacher to (1) give a title* for a new-question set, (2) 
enter the eight multple -choice i£ems, along with (3) correct response 
properly designated and (4) the accompanying definition or informative 
passage. 

An especially useful feature of the DX package is the option of 
selecting the number, of alternatives offered, with items. While all of the 
examples in this manual presume four j cta>ice multiple -choice format, it is 
possible to have the following options: 
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a) or b), 
b) or c) 



a), b), c) or d) .■ * 

\ • a), b), c),- d) or e) 

t) or f) * 
How to select an option is shorn on page (46); If none is selected, the 



program assumes four-choices. 

The second Menu entry .is available for modifying previously written 
question s^Cs. File editing is accomplished through the File Modifier 
program: The program allows ^he teacher to view a file a line at^a time or 
a page at a time. Changes in any line can be made by line deletion, line 
insertion, or line rVplacerent. All necessary instructions are provided .on. 
the screen. \ ' > '* fc • 

Third on the Menu list is a cataloging function. This allows the 
instructor to review the. titles of all question files (and othersl 
available, on the disk. When a given student or group of students is* to be 
tested on a certain unique topic collection, the instructor can peruse/tjie 
available question sets in order to prepare the appropriate examination. 

Selection number 4 on the Menu allows the instuctor to view (or 
print) any established question set. The program utilized for this 
function is called Hard Copy, and cjn be used with or without a printer 
available. Reproducing or viewing a file does not cause it to be altered 



or lost. 



The fifth choice on the Menu is to run the test program. Selecting 

« 

this alternative engages the testing format subprogram (DXOK) and 
automatically initiates the testing procedure. The student's name is 
requested, instructions are delivered, and the first multiple -choice item 



■Y 



is made available for response. Tfce testing program also automatically 

\ , > • . 

stores and interpret%*the test performance, preparing the appropriate 

diagnostic readout. 4 - ■ - * v 

.The final Menu selection available is Quit. Choosing this alternative 

ends the DX package functions and returns the user to' the Apple II 

f • 
computer. (Whatever was the last subprogram used, that's the prograim-now 

in the computer's memory.) 

One of the primary -advantages to utilizing the DX testing system is . 
its ability to create, store^Jand report di a gnosti cutest results. In' order 
to read these results, a separate subprogram within the DX package must be. 
used: Hard Copy. T^iis subprogram is available through^ Menu selection 
number 4. By typing in the student's nane as a file ti£le, this allows 
the teacher to view th§ results on the screen, or to have them relayed to a 
printer if available. 

Although not programs in the true sense, a comment needs to be made 
wi$h reference to the Question Ihd^x and the item files. These are both 
integral, necessary elements ofjthe DX package, and logically so. 
>Certainly, the test package cannot run without first having directions from 
the Question Index on which topics are to be included in the test, and ' 
secpndly, having test files available for use. Both types of files 
(Question Index or test files) are created by means of the Item File W@Br 
and File Modifier programs (Menu #1 and #2) which will be reviewed in 
detail later in Section IV. ' . 

The final subprogram in DX is called €0PY A. By means of this program 
aft unlimited number of duplicates can belhade of any existing DX test 
disk.. You have already used this program to make a backup of the "original 



r ■ 

A 

- . v / 

program disk, if you started at the beginning of this manual. Therefore, 
if you create a test package and want to u^b it in more than ~one computer 
at a time, then it can be -reproduced. Also, teachers or t^st developers 
can share their. D"X test packages and their test files. 

i 

* 





Section V 
* DX in Action: An Example 



The following .section 1s provided to acquaint the prospective DX user 

with^fehe package by reviewing the package and its uses from the beginning. " 

Employing a previously developed item set included on the QX disk, the 

reader will be led through the developmental and implementation phases of 

* • - < / 

the DX system. Simply follow the narrative below and obey all 

instructions. ('You will need an Apple 11+ computer with a monitor «[TY^ 

screen] and at least one disk drive to proceed through this section or any 

sections in this manual' subsequent to this one.) 

STEP 1: Com puter Orientation . 

If you are already familiar wfth the use of the Apple II computer and 
the Applesoft language, you should first make a backup copy of the program- 
disk. Store the original in a safee place. Then, go to "Step 2": Booting 

up the Disk". ' v : - 

If you are not familiar' with the„ Apple II computer, you will find the 

; ! " * ^ 

next several pages very, helpful . They are written to. help you make every 
step of the pnu^dure an easy one. If you find that'you are having 
difficulty, go back atid check the preceding steps to make sure all is 
well. Certain error messages are discussed at the'end of this manual. 

To begin, you should have an Apple II or Apple 11+ with 48K of memory, 
a video monitor and one or -two disk drives controlled by one controller 
card. This is your "hardware". A printer is handy but" optional. Nb other 
hardware is required to execute the Diagnostic Testing System, nor is any 
additional software or programming language necessary/ It is a stand-alone 



interactive package. (Even if you ha vfe -additional hardware, you should 
have no difficulty because the programs do not refer to extra devices.) * 
To make a backup copy of the program disk, which Js your "software", 
1) Insert the program di^sk into the #1 drive. (Do not let f 
\ your fingers tQUch the shiny plastic surfaces which 
t n show through the s^ots-^-the ifatural human oils are 
' sufficient to cadse occasinal difficulties with the 

r 

* v " f 

machine.) * . * . r 

21 Turn on the~^App*le II (via switch is on the rear left of 
- the computer) ancpyour video monitor.^ The* disks will 

whirandyou will see a MENU, which will be used later. - 
Type re", then type "Y"., You will *see a "]", which is 
callfed the ."cursor". > 
4) Type, "RUN COPYA" and press "RETURN 4, and you will see • 

" APPLE DISKETTE DUPLICATION PROGRAM / 
' v . DEFAULT = a ORIGINAL SLOT: 



Press "RETURN" again and you will see 

DEFAULT =1 DRIVE: 1 

Press "RETURN", another time and you will see 



^be^lt ~ 6 . Duplicate . slot: a 



V > / 

Press "RETURN"' a fourth' time and you will see 

v " *, 

DEFAULT « 2 DRIVEN 2 

■ .. .•• . " .. • 

' ' ' If you liave only one disk,' type a "1". If you have two 

disks, .press "RETURN" at this time, appropriate - 

instructions will be given to you by the program. Your *f 

<- » "original disk" is the program djsk supplied with this 

' * . •' 

manual"; your "duplicate^ disk" is. a fresh .blank disk - » 

which will be corns the disk yqu will use^rom here on. 

Before going any further, store the" original program/disk in a safe ^ 

place,, away from he?t an,d moisture^ (You , may. need to refer to it again if 

.: • . 

there^s an unexpected system problem. ) ^ 

Your best bet n'ow is to turn off the Apple II and remove your 
"duplicate disk"--*hich for the remainder 'of 'this, manual is referred to. as 
the bx disk— and take time to read the foil owing 1 chapters and ' y • 
illustrations. The opening chapters will supply back grpun 4 information 
about the diagnostic testfng procedure ; the illustrations will show how the 
Diagnostic Testing System is impleme^fed^fr'om start to finish, while you • 
run the program to see how it works*.' 
STEP 2; Booting up the Disk i 



Again, if you are familiar with the computer, you may sjip this part 

"' \ - ' ! ' / 

and go to "Step- 3: ' The Menu". If you are new to computer^ read this part 

carefully. ; * . 

To,tagin, one must "boot-up" the* computer disk. Booting up a disk is 
Vike priming' a pump— it prepares the- machine^or'its purpose. Booting is 
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accomplished by placing the DX disk into the di-sk drive,* labelled side up, 
while holding it by the end with the labe"h (Some drives require that a ? 
closing mechanism of some sort be operated as *part of the booting process.) 

After the disk is securely placed in the drive, turn on the computer , 
screen, then turn on the computer itself/ As is explained in the Apple 

♦ . . . ■ ■ > . - ■ > * • - ' 

manuals, you will know when the booting process has- been successful because 
you will >hear whirriag, mechanical sounds frQm'your disk drive* Also, mosIP 
drives ha ye a lijjht that indicates ?m operating ^tate. Please, never 
remove or insert a df^c IB to a drive that is still irf the operating state. 
Severe damage may result. 

. STEP 3: Tfe MENU / , , \ 4 ' + 

Type "RUN MENU" and t|ien press "RETURN"^, and the computer screen will 
automatically print out the Menu for you. As described earlier, atr thf s 
point you have several alternatives open to you. The screen lists them by 
shSlring you the following: 

welcome to ucla's \. 
diagnostic test package * >^ . • ' 

From the center for study of evaluation 
los angeles,' california ' ' ^ i 
1 1: create a question set 

2: edit an existing question set 
. 3: list the files on this disk • 
4: list a file to screen or. printer 
* 5: run the diagnostic program 

.". 6: QUIT ' . ., 

" " * ....YOUR CHOICE? . 

r 

To make your selection* simply punch the number key which corresponds 
to the choice you desire. For the sake of^this narrated demonstration, 
just type the number 5. ^ . 
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'-When you make yqur selection you will be' given the opportunity to 
Changs your mind, just in ca*se you goofe'd. the screen, will print: 



...OK? (y/iM^ ,• • , v 

If your Menu choice was the one" you really wanted, jus.t punch the "Y" 
. * ^. ' ' ' ■*''*.', 

key, meaning "Yes". If you had goofed,, or wanted to change your mind, 

punching the "N" key (meaning "No") would let you make a, hew Menu selection 
(try it). - - . 



face "Y" is punched, the disk 3^11' whir again and the screen will 
tell you that the computer is in the process of complying with your Menu 
choicer In our case, the screen would say: 



NOW LOADING THE PROGRAM TO~ 
RUN THE' DIAGNOSTIC PROGRAM 



After a slight delay, the screen would advise you that, the computer 
has been properly programmed by .the disk to fulfill^your Martu choiGe. In 
our example, the screen would inform you that the DX test program is loaded 
and ready to run: 



V 

WELCOME TO OUR . ' . / 

QUICK TEST IN SCIENCE. . . ! - 
...PLEASE TYPE VC-UR FIRST NAME AND 'YOUR /^AST NAME'. 



V 



y * 



✓ 
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. * _ - ■ ■••\ 

' \ 1 : 

; Jhe test*peckage included "with yoiAr DX demonstration disk covers .. * 
science topics* Specifically, thetexam^le test that you will te taking is 
on the topic of "Photosynthesis". Don'tbe concerned if you know nothing 
.^aboivt photosynthesis— the less you know the better this exefjjple will turn . 

out. If you should know something on the/ topic, please follow 
-instructions, not your intuition, thus aiding us in our training and 

< — : % * ' 

familiarization phase. « 

♦ 4 „ .' * • 

As the screen expresses, f the computer is now prepared to encode the 
examinee's name. It. is important to remertber two things at this point: * 
First, the namT^ped in here will be used by the computer as a storage cue 
* for the fcst results. For that reason, it does not really matter how a 
V^c test-taker enters .his or her name, so long as you are aware of the name 
being used/' Later, the teacher uses the satie name as the^name for accessto 
that student' s : results. So, if an examinee types in gibberish, then the 
instructor^ wil V have^ouble later when trying to get test outcome 
feedback. 
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Second, if. the' same name is used a second time with the Identical 

disk,"then. the -computer gets confused. The machine cannot imagine that two 

persons might have identical names, or'that the same person might use this 

.disk on two occasions. If the same name is used twice, the first 

performance for that name is repl aced by the newest performance. " 

" > ' ■ • ~ — ^ 

There are several ways to avoid this .problem. First, one xouTd add 

. * . - • • . J* J < 

the date of the testing session after one's. name, as long as there is only 

one session per date. Another idea is to use a different disk for each 

session. Also, one could number the test occasions by adding an 

appropriate chronological indicatprafter one's name. Generally, however, 
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this, issue is not of any real, .concern, vsi nee the diagnostic readout of the 
• . . • * • ! " '* ' 

test results is usually printed out or recorded soon after each test ' 

occasion. " . - .. • \ • 

^ • - , ^ "II 

After typing one's name, the next, step is^o push the "RETURN "/Key. 
Thf5 is one of the very few times that this is necessary. The "RETURN" key 
is found on the right-h&rid edge of the keyboard. 

The next screen entry reads out as follows: 

" . • a .. . : : 



'...I'M VEFW PLEASED Yo MEET YOU, PETER 
. . . LET jliE GET READY... 



J 



Notice that the computer addresses the examinee' by first name only 
("Peter" in the text example). The machine begins to whir again as it 
records the name and loads the first test in the programmed battery. Oic> 
th4 test *s loaded, the computer gives the appropriate test-taking * 

) ■ ■ " • . .. . 

instructions: 



I WILL ASK YOU A FEW SHORT , 
QUESTIONS NOW ABOUT SCIENCE. 

EACH OF THEM NEEDS JUST ONE ANSWER 

» 

WHEN YOU READ THE QUESTION, 
BI9E YOUR -ANSWER BY TOUCHING 



V 



A, OR B, OR C; QR D 

. . IS THAT OK? 

A. YES, THAT'S OK! 
. .NOW TYPE <A> 
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In this specific example set (photosynthesis), there are four 
alternative choices available with each item presented- The instructions 
clearly orient the student to the use of the four response keys; A, B, C, 
and D* If a test file were structured with more Or less alternatives, then 

4 

the instructions would reflect the choices available. The examinee 
verifies that the instructions have been read by pressing the "A" key. 

NOTE: Due to the complexity of the DX program package, the Apple II 
computer often needs to be coaxed in order to accept a response. If a 
response key is punched and there is a "beep" souno\as well as a screen 
change, then one should simply proceed. However, (if there is no "beep" or 
screen change after a response is made, then the computer has not 
"understood" or accepted the choice made. Simply eyepunch the key until the 
computer responds. Caution your students about this; just tell students to 
press the key they choose until they hear the "beep". 

Once "A" is punched, the test will begin. 
STEP 4: The Test ' 

Immediately, and with no topical cue or introduction, the screen 

presents the student with the first item in the first test topic lattice. 

V 

Figure 2 illustrates the specific item lattice structure for the 
photosynthesis "file. ijlote that the item you see on the screen matches the 
first test item, in the upper left-hand corner (Figure 2): 
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Figure 2 

Lattice Structure of Items from 
the DX Photosynthesis Subtest 



m ARE KAHtfLS INCAPABLE OF MfflTfl- 
SVHTKESIS? 




^^^next question set=== 



• • • ft» 

• ••6. 

• • • C» 
...D. 



THEY LACK CARBON DIOXIDE. 
THEY LACK LIGHT. 
THEY LACK CHLOROPHYL. 
THEY LACK ARYTHROBLASTS. 




4 
> 

HJAT DO 6REEH PLAHTS RELEASE AS A 
IxQEUCT OF PHOTOSYNTHESIS THAT HELPS" 
WIHAL3 LIVE? 
...A. MINERAL DEPOSITS. 
...B. CARBON DIOXIDE. 
,.C. - PROTOZOIC ACIDS. 
...D. OXYGEN * 



X 



AH UPROOTED PLANT HILL QUICKLY DIE; 
EVEN IF IT IS UNDAHAGED. HHY? 
...A. IT LACKS MINERALS NECESSARY 
FOR ERONTH. • 

...B. IT LACKS HATER FOR OXYGEH A?:D 
SUGAR PRODUCTION. 

...C. IT LACKS OXYGEN NECESSARY FOR 
PHOTOSYNTHESIS. 

.'..I. IT LACKS SOIL HITH HORNS FOR 
PROPER AERIFICATION. . « I 

/ - 

HHAT IS THE SOURCE OF THE ENER6Y THAT 
6REEN PLAHTS NEED FOR PHOTOSYNTHESIS? 
...A. LIGHT. 

...B. STATIC ELECTRICITY. 
...C. SOIL MINERALS. 
.D. CHLOROPHYL. 



APOTTEB PLANT LOCKED INTO A CHAMBER 
HITH ONLY OXYi'OI; HATER; AND LIGHT 
KILL SQDH DIE. HHY? 
...A. IT LACKS CAR3GH DIOXIDE FOR 
PHOTOSYNTHESIS.* 

»..B. IT LACKS HICROORGANISHS FOR 
SOIL NINEHAL REDUCTION. ' 
...C. IT LACKS CHLOROPHYL BECAUSE 
THE CHAMBER IS LOCKED. 

D. IT LACKS SUGARS HEEDED FOR 
PHOTOSYNTHESIS. ' jk j 

J ' 
HHAT DO AHINALS PRODUCE DURING 

RESPIRATION THAT'HELPS PLANTS DURIH3' 

PHOTOSYNTHESIS? , v 



rosYp 
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KHSRE CO. PLANTS EET THE CARBON THAT 
THEY HEEDvFCR PHOTOSYNTHESIS? 
...A. CARBOHYDRATES IN THE SOIL. 
...B. AMMONIA IH FERTILIZERS AND 
SOIL. 

...C. HATER. 

...D. CARBON DIOXIDE IN THE AIR. 




KITR06EH. 
OXYGEH. 

CAR90H DIOXIDE. 
TETRACHLORIC CHLOROPHYL. 



AS A RESBbljjf PliOTDSYHTHESIS; PLANTS 
PRODUCE ~ 

...A. HEAT AND OXYGEN. 
...B. SUGAR ftND OXYGEN. 
...C. KIT906EH AND OIYBEhNw 
...D. SUGAR AND NITROGEN. 
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WHY ARE MAMMALS INCAPABLE OF PHOTO- 
SYNTHESIS? 

...A. THEY LACK CARBON DIOXIDE 
. . .B. THEY LACK LIGHT 
...C. THEY LACK CHLOROPHYL 
*...D. THEY LACK ARYTHROBLASTS 



WHAT IS YOUR CHOICE 
(A, B, C, OR D) ? 



Two comments are relevant at this point. First, the DX testing 
package is progammea\ to accept only A, B, C, or D as possible responses to 
this test file. If an examinee accidentally or intentionally presses any 
other key on the ^keyboard, the "what, is your choice" question will be 
reprinted, including the four possible aternative key choices (try it!). 
Once again, the program is fool-proof. 

Secondly, it is important to reiterate that a response key may have to 
be pressed a few times before it registers. 

By consulting Figure 2 one can discern what will happen based on the 
response to this first item. If one gets it correct, then the item set is 
conquered and the first Item of the next topic set will appear. If one 
fails the first attempt, then the first item at. the second lattice level 
will appear. 
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For the sake of experience, It wtn be more Instructive if you would 
please fail the first item. Therefore, PRESS "A". The "beep" is heard and 
the screen adds: * 



SORRY, BUT THE CORRECT ANSWER WAS C 



This message will appear together with the test item failed* for long 
enough to allow the examinee to reread the item and make the. appropriate 
mental corrections. 

Note that the DX system has another built in advantage over pencil- . 
and-paper formats: Immediate feedback. This makes the examination both an 
assessment as well as a learning occasion. 

After a short delay for the feedback, the screen presents the second 
test opportunity- 



■* 

WHAT DO SREEN PLANTS RELEASE AS A 
PRODUCT OF PHOTOSYNTHESIS THAT HELPS 
ANIMALS LIVE? 

...A. MINERAL DEPOSITS. 

. ... B. CARBON DIOXIDE. 

. ..C. PROTOZOIC ACIDS. 
...D. 0XY6EN - 



WHAT IS YOUR CHOICE . 
(A, B, C, OR D) ? 

1 I 

V 



■ > 



Recall that this actually the first chance on the second, or middle 
lattice level (see Figure 2). Again, for the sake of illustration, please 
get this item wrong also. Please PRESS "C". k . 

* 

SORRY, BUT THE CORRECT ANSWER WAS D 

Again, the feedback is immediate and lasts long enough to allow for 
encoding and self -correct ion . , 

The next item to appear is the first chance at the lowest difficulty 
level of the lattice (see Figure* 2): 



r 



0' 
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WHERE DO Pl/ANTS GET JHE CARBON THAT 
THEY NEED FOR PHOTOSYNTHESIS? 
...A. CARBOHYDRATES - IN THE SOIL. 
...B. AMMONIA IN FERTILIZERS AND 
SOIL. 

...C. WATER. ^ y 

...D. CARBON DIOXIDE IN THE AIR. 



WHAT IS YOUR CHOICE 
(A, B, C, OR D) ? 



Again, for the sake of instruction, please make the wrong response, 
Please PRESS "B". 



SORRY, BUT THE CORRECT ANSWER WAS D 
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At this point, the examinee has failed to pass any item. However, to 
^say that three errors have produced a score of 0 (zero) is to oversimplify" 
the real diagnostic results. Actually, the student has failed to respond 
correctly to analogous content questions from most difficult to easiest. 
Obviously, the s.tudent need some instruction or refreshing. Therefore, 
with no further prodding the computer "Will provide a simple statement or 
definition to any examinee who fails at the easiest lattice level (see 
Figure 2): 

PHOTOSYNTHESIS IS THE PROCESS BY 
WHICH GREEN PLANTS SYNTHESIZE OR 
PRODUCE THE SUGAR THEY NEED FOR 
ENERGY. LIGHT; CARBON DIOXIDE; AND 
WATER COMBINE IN PLANT CELLS TO 
PRODUCE SUGAR AND dXYGEN. 

r • 

WHEN YO'U ARE- READY, TOUCH ANY KEY... • - 

The test developer can. place any information desired in the definition 
position in the test file. For this specific file, we chose a simple 
explanatory statement about photosynthesis. The examinee informs the 
computer that the definition has been encoded, and that; s/he is prepared <to 
resume testing, by pressing any key. 



The next item to appear is the second chance at the-easiest level: 

AS A RESULT OF PHOTOSYNTHESIS;, PLANTS 
PRODUCE — , 
...A. HEAT AND OXYGEN. 
...B. SUGAR AND OXYGEN. 
...C. NITROGEN riND OXYGEN. 
...D. SUGAR AND NITROGEN. 



WHAT IS YOUR CHOICE 
(A, B, C, OR D) ? 



Based on the definition you read, you either learned the answer to 
this question, or your memory was refreshed. For the sake of our exercise, 
let's get this one RIGHT. Please D RESS W B" . 



THAT'S RIGHT: GOOD JOB! 

Note that the screen now provides reinforcement as feedback* The 
praise remains for a brief tine. Then, the second test on the middle le\el 



occurs (see Figure 2): 



WHAT IS THE SOURCE OF THE ENERGY THAT 
GREEN PLANTS NEED FOR PHOTOSYNTHESIS? 
...A.' LIGHT 

.'. « B. STATIC ELECTRICITY. . 
. . .C. SOIL MINERALS. 
...D. CHLOROPHYL. 



WHAT IS YOLP^CHOICE 
(A, B, C, OR D) ? 



For the sake of our instruction, please goof again. Please PRESS "D" 



SORRY, BUT THE CORRECT ANSWER WAS A 
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. At th-is point our nythical examinee has failed a second time at level 
2 of the lattice, the DX program interprets this to mean that further 
^opportunities at any level above level, 1 would not clarify the diagnosis of 
comprehension level, and 'so the topic set ends ajt this point. The student 
is simply told: t 

. . .OK, PETER ' 



A Mor 



...HOLD ON FOR JUST A MOMENT, PLEASE... 

. w + 

■* 0 

• , X ■ 

While this message appears, the disk drive whirs as it loads the next 

test set. In our example there is no "next test set", and so the computer 

advises the student thatrthe test session is over: 



THANKS, PETER 

I'VE ENJOYED WORKING WITH YOU TODAY. 
...THAT,'S ALL FOR NOW. SO LONBX- 1 ! 



. ) 
■ / 



\ 



This message will always appear^after all test sets in the Question 
Index have been. completed. r^fCNsimply .indicates to the student that s/he is 



ERIC 



free to leave. , 1 



STEP ,5: diagnostic Output , . 

While the student has been taking the tests, the computed has been 
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^recording the response patterns and (2) preparing the diagnostic * 

interpretations. In our example test, the computer should have recorded 

one error at level 3 (highest level), two errors at level 2, and one/error 

and one success at level 1 (the lowest level). Also, there should be a 

report made that we" reached the definition. 

In order to access and read the diagnostic output, one needs to return 

to the menu. -After the farewell greeting which finalized the test 

■"■■* ■ ■ »> * 
occasion, the Apple II "cursor" appeared at the bottom of the screen. The 

* ■ 

cursed is a left-handed bracket whidh is flashing on the lower left side of 
the* screen. To return to the menu, simply type RUN MENU. The screen will 
look like this: ' 

* 3 RUN MENU 

Be sure to leave a space between the two words. Use no punctuation; 
that will only confuse DX. After typing that statement, press the "RETURN", 
key. Whirring will soon be followed by a reappearing of the ISJwni on the 
screen. 

(Note: If an error message appears when you press the RETURN key, 

/ - - . • 

then simply retype RUN MENU a- second time, being extra careful of spelling 

and spacing. Then press "RETURN". 'If that doesn't work t (1) turn the 

computer off by reaching in back on the Teft side, (2) wait a couple of 

seconds, and (3) turn it back on . The MENU will automatically appear.) 



\ 



WELCOME TO UCLA'S 
DIAGNOSTIC TEST PACKAGE \ 
FROM THE CENTER FOR STUDY OF EVALUATION 

LOS ANGELES, CALIFORNIA 
1: CREATE A QUESTION SET 
2: EDIT* AN EXISTING QUESTION SET 
3: LIST THE FILES ON THIS DISK 
4: LIST A FILE TO SCREEN OR PRINTER 
5: RUN THE DIAGNOSTIC PROGRAM, 
6: QUIT *- 
YOUR CHOICE? 



This tine we wa^the computer to produce the results of the test 

experience. These results-are currently stpred as a file named for the 

... \ 

person tested. To access that or any diagnostic file, choose MENU item #4 
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When asked to verify, your choice, type n Y 4 \ 
1 ...OK? (Y/N> ' 

Remember'°to press the "RETURN" button (if nothing happens). 
The screen will now advise you that ityis getting the^file reading 
program, ready: 



NOW LOADING THE PROGRAM TO 
LIST A FILE TO SCREEN OR PRINTER 



When the computer .is ready, it will ask you to supply the name of the 
file you want to readT" Here, it is important to learn tow things. First, 
all files of ts.t results are ear-marked with the DX +»g. So-, for example, 
Pete's (Pete Matheson) test output would be stored as DXPETE MATHESON. 

In other words, it is critical to know the name that the test taker 
i/sed. Then, just add a "DX" to the front of the name WITHOUT A SPACE. 

The screen looks like this when it wants you to name the file desired: 



. . . EXAMINING NAME? 



Touch no keys except those* needed to type the file name. Once the file 
name is complete, press the "RETURN" key. 

.♦ The computer will then ask whether you want the file results printed. 
If you say "no", then the screen will show you the file. If you say "yes", 
the screened the printer (if available) will both produce the "results. 

. \ ■ . . v. : - - 

DO YOU. WANT PRINTED COPY? (Y/N) ( 

If you get a .message stating that the file you requested does not 
e xi st, 



# FILE NOT FOUND 



several things could be wrong. First, you may have made a typographical 
error. Simply 'retype the correct file name (with "DX") and try again. If 
that fails, then there may be a discrepancy between the name used arid*your 
expectation. If this is possible, then there is no other recourse but to 
examine the list of file names. 



This is done by pressing/the "RESET" key. A cursor will appear.^ Now 
type^ RUN MENU, followed by pressing the "RETURN" key. When the menu f 
appears, make selection .number 3. • u 

' 3: *LIST THE FILES ON THIS DISK 

# . ■ 

After you verify your choice, the screen will produce a catalog 
listing of all files on the' disk you have booted up. Read^he li|t in - 
search of the appropriate file desired. Note that the list. may be longer 
than just one page of the screen, so use the space bar to continue from one 
page of the catalog to the next. 

When yoti spot the desired file, jot down the precise name.. Then, 
press the '"RESET? key. Type in RUN 'MENU, and proceed With option #4i 

If you said "Y" (yes) to the request conce'rning'printing, the screen 
will now remind you to turn the printer on: 



...MAKE SURE YOUR PRINTER IS READY 




\l"hen, suddenly, the diagnostic printout appears. The first line 

repeats the ftile name (the examinee). The second line lists the title of 

-* .</ 
the first subtest administered.. The third line lists the response patern. 

The fourth line provides the diagnostic feedback. The three-line group of 
test name, response pattern, and diagnostic narrative will be provided for 
ea"ch test taker at a given occasion. 

Your feedback, if ytou followed the response instructions, should look 
like this:- * ♦ ' 



. PHOTOSYNTHESIS *> 
X..XX.XP. ' x ' 

PASSED AT LEVEL 1 AFTER REACHING 
xfHE DEFINITION" DIRECTLY AND THEN PASSING 
j AT LEVEL 1 BUT FAILING AT LEVEL 2 

# OF LINES IN FILE =4 



To Interpret the response' pattern, just remember that there are three 
possibilities at each lattiQe level. For each test item attempted, "X" 
reports a failure and "P" reports --a success. Dots (•) designate items not 
encountered by the examinee* , ■ 

In our fase, divicfe the perfofmance patted into thirds by adding 

slashes^ V ' \ 

X../X)h/XP. 

o.' - 
The first section corresponds to the three high-difficulty items. . 

Only one" was attempted (the first item given.) and it was failed. The 

4 \ 

middle section corresponds te-the three middle-difficulty items. Here the 
examinee had two chances, and failed both 1 . Finally, the last section 
applies to the two low-difficulty items and the definition (always reported 
as a dot, whether seen or not). ■ l The pattern shows that the first try at 
this level was a failure, but the second was a success. js 

Note that the only way one can reach the^second item at the lowest*' 
level is by first seeing the definition (see Figure 2 U* For that reasoh, 
no entry is made in the pattern indicating that the definition was reached. 
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To interpret ^the diagnostic readout, two things must be considered. 
First, the beginning 'of the narrative will designate the lattice level at 
which the test was passed. * Also, if the definition is reached with 
successive failures at all three levels, then the pattern is reported as 
"reached the definition directly/V or "by the most direct route". Second, 
the narrative is not topic specific, but is designed to be applied to any 
test utilized"^ * ' " 

Summary ' . ; # . 

Take' some time and run through the. test a jfew times. Produce, a ,. 
different pattern each time— there are 18 in all. Be sure to change your 
name each time to avoid problems.: , 
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Section VI ; . % I * 

Do It Yourself 

Having experienced the trial runs in Section IV, you aire now prepared 
to generate your own test file* This, is accomplished in three simple 
steps. 

STEP U Item Generation 

Based on the logic of the lattice structure already discussed, choose 
a topic which can be easily adapted to three levels of difficulty. When 
you get more practice, any topic will work for you. For now, though, 
concentrate on an easy topic and obvious taxonony. 

Next, write ..out the central concept to be assessed and definition upon 
which all of the items win be based. Then, using the taxonony you decided 
upon, generate eight items: 
o two most simple 
, o three middle difficulty 
o three highest difficulty 

Having designed the item with four alternatives each, you are prepared 
to create a new test file. 
STEP 2: Test File Creation and Editing - 

Once the test items and definition have been generated, place the DX^ 
disk in the disk drive and turn on the machine. As usual, the Menu wilT^ 
automatically appear'. For Item file creation, we are interested in the 
first two Menu choices: 
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1: CREATE A QUESTION SET 

2: EDIT AN EXISTING QUESTION SET 

► 



Following the same selection procedure we have used before in the Menu 
program, select option #1. 



, YOUR CHOICE? i 

...OK? (Y/N) 



Soon the computer will acknowledge your selection and ask you to name 
your new file. It's probably best to en title^Tt^ based on the topic 
covered, al thou gty any title you can recall will suffice. 

After carefully typing the file title, press the "RETURN" key. The 
screen will fill up with instructions on how to properly use the FILE 
WRITER subprogram. After you read the instructions, begin typing the 
first, most difficult item. The actual sequence for entering the items is 
as follows: 

HIGHEST LEVEL 12 3- 

MIDDLE LEVEL 4 5 6 

LOWEST LEVEL 7 8 

DEFINITION 9 

1 

T he items must be entered in that order for DX to work properly! 

\ 



( 



EMC . / . ..12 



, There are a couple of limits helpful in using the FILE WRITER. If you 
want to indent, just place a period (if) at the beginning of the appropriate 
line and then you can space inwa'rd. If you just use the space bar without 
a period, the computer will ignore the indenting and print the items 
without it. 

Also, if you should try to use commas, colons, or any punctuationother 
than periods or slashes, the computer will ignore them. (The reason for 
this is that the programming ifi DX is~already complex. Increasing the 
complexity by programming for punctuation, underlining, or quotes will only 
decrease the program's efficiency and create potential malfunctions.) 

The question mark (?) is the new cursor. Type the text one line at a 

8 

time. Tf you have a sentence to type, limit it to 40 elements (spaces and 
letters) or less per line by using the "RETURN" key after each line. Too 
many letters per line will cause the computer to cut words in half when it 
runs the test, making it hard to read. Also, start each response 
alternative on a new line. J 

After each item is entered, along with its corresponding response 
alternatives, it is CRITICAL to indicate to DX which answer is correct. 
This is done by typing an asterisk, followed immediately by the letter of 
the correct answer^ <^ 

A sample item, complete with all. alternatives and the correct answer, 

might look like this: 



?HOW OLD IS ALBERT? % 
7. A. 13 \ 
> B. 1? | 
7. C. 35 
?. D. 188 
?*C -, 



33 



- 40 - 



Three more hint?. Don't number your Items. If placed In* the proper^ 
sequence, the DX system will know what numbers they are. Don't use an 
or an "*" in the first column except for the special purposes noted here. 
It will confuse your computer. Also, don't be concerned about small 
errors— you'll have a chance to correct them later. . 

/After the last item, enter the definition by simply following the same " 
40-letter-line rule, as well as punctuation limitations Definitions 
should be kept as brief as possible--you have limited screen room (about 15 

lines). ' . 

Once all items and the definition are typed out, end the file with a 
line of three asterisks (***), followed by^hne with one exclamation 



poi 



nt (!). This is critical because these two lines of punctuation mark 



are what indicate to the computer that you are through, and that a file 
ends at that point. 



< 
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1 HOW OLD WAS ALBERT? 1 * 

2 '. • . A 15 

3 . B. 19 

5 . D. 1,88 

6 *C 

7 SEVEN MORE ITEMS 

8 ...AND ONE DEFINITION STATEMENT. 

9 *** 
10 ! 



After the exclamation po>rrtppi«ess the "RETURN" key once more -and the 
screen wfll ask: 

WOULD YOU LIKE TO REVIEW "> Y 

Typing "Y", followed by "RETURN", will cause the entire file to go by 
On the screen so that you can proof it. Note- that the lines are now 
numbered for your convenience. 

After the review is complete, the screen will ask: 

V » ^ , 

. . . ANY CHANGES ? 

Q 

-> . 4 

The program called the FILE MODIFIER is a more extensile editing 
program. For that reason, we advise that you save the^file, errors and 
all, by indicating that you do not need any changa. , 
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To the next two questions, tell the machine that you cfesire no more 
reviewing ("N"), but that you do want to saw the file ("Y"). 

.....DO -YOU WANT TO REVIEW ONCE MORE ? N 
.. .SHOULD THIS FILE BE PRESERVED? Y 
...NOW SAVING ALBERT ON DISK 
...WOULD YOU LIKE PRINTED. COPY? N 

J At this point, on the cue of the cursor, request" the Menu again: 

" 1RUN MENU 

This timSj^select option #2 in order to edit the file you created. 
After verification ("Y"}, the screen will print: 



f 

NOW LOADING THE PROGRAM TO 



EDIT AN EXISTING QUESTION SET 



* When asked what name, type in the name you chose when youscreated the 
file. Then, press "RETURN". 

...WE ARE MODIFYING * 
AN EXISTING FILE*. . . 

NAMED WHAT ? ^ 

...ON WHICH DISK <l> OR < 2 > ? 

When asked which disk drive is being used, generally #1 is the 
^answer. The next messages alert you that the file is being (1) loaded and 
(2) duplicated. TJie duplication process (back-up) makes it impossible to 
lose tne file completely, even .if you should goof during editing. 



NOW INPUTTflNB. . . 

NOW MAKING A BACKUP COPY 



The text of the file will appear 20 lines at a time. If you want to 
make any changes, simply type "Y", press "RETURN", and then the line 

number, press "RETURN". The computer 'then produces a list of editing 

. • v 

instructions which involves three options: 



WHICH LINE # ? 

IF YOU LIKE THE LINE, HIT <SPACE> 

IF YOU WANT TO REPLACE, TYPE <R> 
THE* TYPE THE NEW LINE... 

V . 

IF YOU WANT TO DELETE THE LINE, TYPE <D> 



At the bottom of the screen, the erroneous line is reprinted, and the 
question nark (?) cursor appears. Indicate the letter for the editing 
function desired (R, I, or D), press "RETURN", and follow the appropriate 
instructions. For "D", the line will automatically disappear. For "R", 
you will need to retype the whole line. For "I", the computer will allow 
you to insert a* line of text ABOVE the line number you told it to change. 
If you decide not to change tflfline at all, press the space bar and no 
change will .be made. 



If no changes are cfesired for the lines being viewed, then type "w" . 
for "atfy changes?", and press "RETURN "--the next 20 lines will be listed. 

In this manner one eventually reaches the epd of the file. At this 
, point a >fc§w set of questions appear one at a time: 

...ANY CHANGES ? 

/ 

ft • h 

' ■ i > 4 ! 

"Y" sends the entire process back to the firsts item again. "N" brings 
tfoe next question: 



■DO YOU DESIRE TO ADD LINES ? 



"Y" produces the cursor (?) and allows you to add new lines to your 
file. "N" produces the next inquiry: '• 

SHALL WE PRESERVE THIS FILE? Y 



"N" returns to the first of these three questions. <If you are 
thoroughly displeased with the editing you have done, don it try to go back 
to edit your editing. Instead, press "RESET" and return to the MENU, 
select option #2, and bgin again as da scribed at the middle of page 42.) 
Remember, however, that the back-up copy still exists. (Consult the 
catalog— Menu option #3.) "Y" produces this output: 



YOU NEED TO CHOOSE: 
<R> FOR REPLACEMENT OF CURRENT FILE* 
<N>*FOR NEW FILE INSTEAD 



_ . ■ « x ; 

Since you are editing .an existing file, type "R", *rtd press "RETURN". 
This will cause the new edited version to replace the old version which had 
the errors. 

STEP 3: Question Index 

—The computer administers the files that are placed in the Q"estibn 
Index. In,order to test your new file, you need to register it in the' 
Question Index file. To do this, you simply use the FILE MODIFIER 
subprogram. ' 

Either turn the machine on anew, or type RUN MENU after the cursor. 
When ttje menu appears, select option #2., verify your choice ("Y"). 

When asked to name the file cesired, type QUESTION INDEX, then press 
"RETURN". When the file appears, it will read as follows: 

1. Photosynthesis w 

. • /• > .. . . ■ • • 

When asked whether you want to make any changes, you have two 
choices. First, you may want to simply add your file' to the test setA In 
that case, "I" is the test choice. By inserting your file above line #1, 
it will appear first in a two-test exam. By inserting it above line #2, it $ 
will appear second in the exam. 
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The second\al tentative is to remove Photosynthesis and put your file 
alone ip_Lts^p1ace . This best achieved with the "R" . option. Simply say 
that you want changes ("Y"), in the first line ("1"), and then replace that 
Ttne with the "R H command. When the cursor, (?) appears, type in your, 
test's title. The new Question Index will appear with the change. 

Note that removing Photosynthesis 7 or any test^file from the Question 



Index does not erase or destroy the file. It can be replaced later if 

desired. A 

Make sure that the last line of Question Index 4s still an exclamation 

point (!). Without thai}, the DX system will not work. 

After these changes, follow the same file preservation procedure you 

used after editing- your test file. Be sure not to rename the Question 

Index. . „. ' 

„ ^ The DX system is now ready to run. Enjoy your new test file . 
-NOTE : An Option for Number of Multiple-Choice Alternatives. 

The program assumes that the multiple-choice test uses form items. 

But you can tell it differently if you need to. In the QUESTION INDEX, the 

last line is "!". If you desire to have any of the following combinations; 

use the matching code instead. 

Need Use 

Two choices a) or b) — . — — "12" 

Three choices a), b), or c) 4----— ~- — - — "13" 

Five choices a), b), c), d),"ore) H !5 M 

True -false ■ 11 



ERIC 
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.You "can also turn off the feedback which Informs, the student whether a 
given response is correct or Incorrect, if you desire, fferely add the 
letter "N" to the end of the lines above. "!2N" would mean H a) or b)" 
without feedback. 

Summary , • ' . v 

Ydu can now construct tests which consist of any series of topic 
subtests that you have store. By restructuring the Question Index, you 
control the question sets that will be used and the order in which they 
will appear. Tests can be geared to individual or group jheeds and 
restructured at any time. Remember, the DX feedback automatically reports 
tb^exact topic titles which a. student has been tested on, so you don't 
need to remember old Question Index contents. 



\*' ' Section VII . • * 

^ - Diagnostic Feedback • </ 



The- DX testing program automatically produces three types of feedback 
as a result of each test participation expe rt^nceT^ In Section IV, Step 4,, 
the procedure for obtaining a printed version of the feedback was 
reviewed.. Also, that narrative outlined the nature of the three -facetted 

feedback output. This section of the manual explains soma of the 

interpretations associated with the possible test-performance |»ths, as 
well as some brief comments qn scon ng procedures applicable to the DX 
feedback. 

Feedback Interpretation * 

As was explained earlier in the manual (Section V, Step 5), there are 
1& different paths possible in the eight item, lattice used in the DX system 
(see Figure 1). Of those pathsl ten are completed without the student e\erj 
reaching the definition.^ Interpreting results from these—ten non- 
^definition rputes is^rather straight-forward; all performances are based on 
knowledge attained prior to the test experience, and the resihts are a good 
indication of the level and depth of the students unprompted 
comprehension. 

An interesting issue arises, however,, when a student follows an error 
path which provides the definition. Basically, if the examinee \contintes 
the test after encountering the definition, then IjHs difficult to 



interpi^t the results. 

For example, if a' student had-rnTpri or knowledge of a given topic, but 
was an excellent reader, then reading the definition mi girt be sufficient , 



for immediately answering questions. Subsequen\passing or failing of 
items is not necessarily an indication of long term knowledge or classroom 
learning of the material, but rather it becomes a reflection of reading 
ability and short term recall. That interpretation is not readily 
available through DX. 

The only real answer to this potential Vo 0 ! 6 " 1 H* s Wltn the teacher 
or fee'dback interpreter. As is true with any test results, no single score 
performance should ever be interpreted in isolation. (Pophanu 1981). Check 
relevant results from other tests. Compare reading scores. Also, compare' 
scores on similar topics involving other item sets or contrasting 
definition statements. Finally, retest the studeift. If the concepts are 
truly mastered, then the performance on subsequent testings or related 
^materials will reflect similar patterns. 

Another potential^danger in the feedback stems from the nature of the 
DX system. In order to make it easier to assessNS series of topics, each 
separate test experience was designed to Identify students who definitely 
know the material with minimal time expenditures. Repeated item' trials for 
such pupils would only waste time and increase the probability of chance 
errors. For these reasons, a correct response to the first item is 
interpreted as mastery. The problem is that the/e is a chance that a naive 
or non-mastery student may get the first item right by just guessing! This 
would produce completely misleading output. 

There are a couple of efsy steps which, can be taken to reduce \he 
probability of that occurence. First, use as many distractor alternatives 
as possible in your test items. The more sound alternatives to choose 
from, the less the chance of guessing correctly. Secondly, be sure to put 
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the most challenging item as the first in the lattice, thereby providing a 
second ""fail -safe against guessing. ( . 

Scoring Procedures , ' 

' The different types of feedba'ck generated by DX offer. a variety of ; . 

scoring possibilities. The most obvious is^the traditional "number 

correct"* computed by adding up, the. number Of "P" {correct response, or _ i 

m P»ss m 1 "scores from the feedback. This is, however, a potentially 
• • •' ".••«. 

misleading score, because a student who knows the material well will only* 

get one "chance to even score- correctly before bouncing out of the lattice." * 
Meanwhile, a student linacquaintej^th the topic will j^ve more 
-opportunities to score correctly and to obtain a higher totaj score. 

A better scoring ^method would be to report how many items a given 
examinee got right out of those s/he had the opportunity to answer. In 
- statistical jargon, this 1s the proportion Correct, or in simple language, 
> percent correct. To. calculate this, first count how many items in the set 

the student was allowed to answer, whether s/he got them right or wrong, 
pjfext, count how many s/he got right. Fkrm a fraction by placing the number 
of righ,t answers over the total number of attempts. Perform the division 
and you'll. get a proportion, which is a decimal between zero (0) and 
positive one. (+1 00). Then^ just slide^the decimal point to the right two- 
spaces , and now you have the percent correct. 

For example, imagine Barbara took a history test oh the DX. Her SCORE & 

» , ' * • * ■ • " 

pattern from the diagnostic readout looked like this: .„ ^ 

XXPPPXP. 
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In this pattern, Barbara got seven chances to answer. She' got four right. 
Hbr proportion score, then is: 

4 = .5714 

T • - 

Moving the decimal, her percentage score becomes: 

57.14% ■ 



Another useful scoring procedure involves identifying and reporting 
the number corresponding to the lattice level at which a student passed. 
So, for example, a student sqoring at level 3 would be higher than one at 
level 2. One could add up the levels across all subtests and get a 
composite score of sorts, or take 1 an overall average across the subtest— 




Section VIII 
Conclusion 



Repeated experiments have shown the DX system to be reliable and 
useful at all levels of school testing. .The feedback has been especially 
exciting for classroom teachers, whoSiave used the package. Students have 
been more positive about the prospect of being tested by computers than by 
pencil -an d-paper measures. 

Again we add that DX is not a panacea for the testing world. If used 
correctly, however, the DX philosophy and "accompanying software promise to 
improve testing in classrooms and provide comprehensive feedback for 
teachers and administrators. 



